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After a brief introduction on community colleges as a pathway to higher education and earnings, 
I describe a few approaches to evaluating the effectiveness of policies and programs designed to 
benefit students and issues to consider in determining standards of evidence. I then present three 
examples of programs that have been carefully studied and conclude with thoughts on bringing 
programs to scale. While there is much to be done to improve student outcomes there is also 
reason for optimism. Many states and colleges are piloting reforms, and there is a growing body 
of evidence on strategies that work. 

Community College as a Pathway to Higher Education and Earnings 

According to the most recent data from the U.S. Department of Education, community colleges 
enrolled 6.7 million students in 2007-2008 — or more than one-third of all students enrolled in 
higher education institutions. 1 In part because of their open admissions policies and relative low 
cost, community colleges enroll larger percentages of nontraditional, low-income, and minority 
students than four-year colleges and universities. A primary reason why people pursue a college 
education is to boost future earnings. Over a lifetime, a worker with an associate’s degree will 
earn nearly $500,000 more than someone with no education beyond a high school diploma. 
Individuals who earn a bachelor’s degree will do even better, earning roughly $1.1 million more 
than someone with an associate’s degree and $1.6 million more than a high school graduate. 3 

Anticipated future earnings are not enough to hold students in school or ensure progress. 
Longitudinal research by the U.S. Department of Education indicates that six years after entering 
community college, only 23 percent of degree- seeking students had completed an associate’s 
degree and 13 percent had completed a bachelor’s degree. An additional 17 percent had not 
earned a degree but were still enrolled at a college or university. Note that these figures capture 
enrollment at any institution, not just the community college where students began their studies. 4 

The low success rates among community college students are due to many factors. A major 
problem is that a majority of students require developmental (or remedial) course work in English 
or math before they can go onto college-level courses. Unfortunately, pass rates in 
developmental course are extremely low; for example, two-thirds of students assigned to 
developmental math never complete it. 5 A second problem is that most community college 
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students have work or family obligations that compete with school. Research shows that part- 
time attendance in college, 35 hours or more of work each week, and responsibility for 
dependents are among the major “risk factors” associated with low persistence and completion. 6 

Approaches to Evaluating Effectiveness and Determining Standards of Evidence 

Over the past decade, efforts by government, regional accreditation agencies, and foundation-led 
initiatives, such as Achieving the Dream, have made community colleges aware of the need to 
pay closer attention to students’ academic performance and progress toward degrees. Most 
community colleges track basic measures like retention and graduation, and some even gather 
information on students who transfer to other colleges or enter the labor market. Such reporting 
is important for monitoring institutional performance and setting goals for improvement, but is 
not sufficient for evaluating the effectiveness of any particular policy or program. Community 
colleges are complex organizations, and many factors may affect how students perform. External 
conditions also matter; the current economic recession, for example, has led to a surge in college 
enrollments but probably has made it harder for recent graduates to find jobs. 

To evaluate the effectiveness of a community college policy or program, it is essential to 
introduce a counterf actual — that is, some means of determining what would have happened if 
the policy or program did not exist. To illustrate, suppose a community college implements a 
new advising system in order to improve student retention. For evaluation purposes, it is not 
enough to know that 70 percent of students who went through the advising system were retained 
the following year; quite possibly, students did just as well without it. The evaluator’s job is to 
find an appropriate comparison group to estimate the “value added” of the new advising system. 
If similar students who did not get the advising were retained at a rate of 50 percent, then the 
value added or impact of the program is the difference between the two retention rates (70 
percent - 50 percent = 20 percentage points, or a 40 percent increase in retention). 

In situations where a policy or program is being introduced as a pilot — or when there is 
insufficient capacity to serve everyone — most evaluators agree that the best approach to 
measuring effectiveness is the randomized control trial. A group of individuals that is targeted 
for an intervention (such as the new advising system mentioned above) is sorted into two groups: 
a program group that receives the intervention or a control group that does not. The sorting is 
done using a lottery-like process, so that every individual has an equal chance of ending up in 
either the program or the control group. The strength of a randomized control trial is that it 
ensures that the composition of the two groups is virtually identical at the beginning of the study 
— not only in observable characteristics, like age and gender, but also in unobservable 
characteristics, like motivation. By tracking both groups over time and comparing their 
outcomes, researchers can be confident that any differences are due to the intervention — and 
not to one group starting off more advantaged than the other. 

Randomized control trials require skill to implement well and are still uncommon in 
postsecondary education research. More often, researchers will try to construct a comparison 
group that resembles the program group as much as possible on observable characteristics, like 
age and educational history. For example, a group of freshmen who participated in a new 
advising program will be compared to a similar group of freshmen who did not participate. The 
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weakness of such designs is that the two groups may differ in ways that are not readily observed. 
This is a particular concern if the program was voluntary and served students who were already 
motivated to succeed. Because matched comparison group designs can never completely remove 
the possibility of underlying differences between the program and comparison groups, the 
evidence that comes out of such studies is not as reliable as that from a randomized control trial. 

Neither randomized control trials nor matched comparison designs are well-suited to policies or 
programs that affect an entire college or population: for example, a new requirement that all 
entering students attend an orientation to inform them about college procedures and resources. In 
these situations, evaluators may try to compare outcomes before and after the policy or program 
was introduced. The best designs to evaluate such policies, known as interrupted time-series, will 
collect observations on an outcome of interest at many points before and after the policy went 
into effect to determine whether a significant change occurs after the point of policy 
implementation. Unfortunately, such designs can never eliminate the possibility that undetected 
changes in student composition or the environment may also affect the trends. 7 In sum, 
randomized control trials produce the best evidence that an intervention caused a change in 
outcomes. They are not, however, the only approach to evaluation, nor are they feasible in all 
situations. 

Examples of Promising Programs that Have Been Carefully Studied 

In the mid-2000s, MDRC conducted randomized control trials of several programs designed to 
improve student outcomes at community colleges. The studies — which fell under the umbrella 
of the Opening Doors Demonstration — were supported by a consortium of foundations, the 
U.S. Department of Labor, and the U.S. Department of Education. 

One of the Opening Doors programs consisted of a Learning Communities intervention at 
Kingsborough Community College in Brooklyn, New York. The Learning Communities targeted 
incoming freshmen, the great majority of whom required developmental English. Students in 
Learning Communities were placed into groups of 15-25 that took three courses together: an 
English course geared toward their level of proficiency; a regular college course like 
introductory psychology or sociology; and a student success course, taught by a college 
counselor, that covered effective study habits and other skills necessary to succeed in college. 
Laculty who taught in the Learning Communities were expected to coordinate assignments and 
meet periodically to review student progress. The idea was to build social cohesion among 
students and faculty and to help students apply the concepts and lessons across the courses. 

More than 1,500 students participated in the Learning Communities evaluation and were, as 
noted, randomly assigned to either a program group that participated in Learning Communities 
or a control group that took regular, unlinked courses. The students were young (mostly 17 to 20 
years old), low-income, and highly diverse in terms of race and ethnicity. The research team 
tracked program and control group members for two years and found that students in the 
Learning Communities were more likely to feel integrated at school and be engaged in their 
courses. They also passed more courses and earned more credits during their first semester, 
moved more quickly through developmental English courses, and were more likely to take and 
pass an English skills assessment test that was required for graduation. It is important to note that 
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these effects, while statistically significant, were generally modest. For example, after four 
semesters, students in the program group earned an average of 33.2 college credits, compared 
with an average of 30.8 credits for the control group (a difference of less than one course). 
Moreover, contrary to expectations, the Learning Communities did not have an immediate effect 
on persistence. Kingsborough is only one test, however, and a new set of randomized control 
trials is underway in five states to build more evidence on this type of program. 

Another Opening Doors study in Louisiana tested the effectiveness of a performance-based 
scholarship targeted to low-income parents who attended two community colleges in the New 
Orleans area: Delgado Community College and the Louisiana Technical College-West Jefferson. 
The scholarships offered $1,000 for each of two semesters ($2,000 total) if students stayed in 
college at least half-time and maintained a “C” or better average. The scholarships were paid in 
increments at the beginning, middle, and end of the semester, and program counselors monitored 
students’ academic performance. Eligibility was limited to students who were parents and whose 
household income was below 200 percent of the federal poverty level. 

A little more than 1,000 students enrolled in the Louisiana study, mostly mothers in their 20s 
with one or two children. Half were randomly assigned to a program group that was eligible for 
the scholarship, while the other half were placed in a control group that was not eligible. Both 
the program and the control groups continued to receive federal Pell grants and other aid for 
which they qualified. The evaluation found that performance-based scholarships gave students a 
substantial boost. For example, students in the program group were more likely to register for 
college and attend full-time, even though only half-time enrollment was required to receive a 
scholarship. They were also more likely to stay in college. In the second semester of the 
program, 65 percent of the program group registered for courses, compared with 50 percent of 
the control group. And, finally, students in the program group completed more course credits 
than those in the control group, earning on average 3.5 more credits (a little more than one 
college course) over four semesters. 9 Similar to Learning Communities, a new set of randomized 
control trials is taking place in six states to build more evidence on performance-based 
scholarships in other settings and with other types of students. 

A good example of a matched comparison design is provided by an evaluation of Washington 
State’s Integrated Basic Education and Skills Training (I-BEST) program, conducted by the 
Community College Research Center (CCRC). I-BEST aspires to help community college 
students gain proficiency in English and math and also prepare them for specific occupational 
fields, such as nursing, early childhood education, and automotive repair. What is unusual about 
I-BEST is that the English and math instruction is integrated into the occupational curriculum, 
rather than the more conventional approach of teaching them separately. For example, students 
in nursing programs attend English classes that emphasize medical terminology and writing used 
in health care settings; students in automotive repair learn how to read manuals and use 
instruments needed to diagnose and correct engine trouble. 

Researchers compared the academic outcomes for 900 I-BEST participants with those of more 
than 31,000 students in regular developmental education courses who were matched on the basis 
of demographic characteristics, educational background, and enrollment patterns. The study 
found that I-BEST students had higher persistence rates, earned more credits toward a college 
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credential, earned more occupational certificates, and showed greater improvements on tests. 
Because it was not a randomized control trial, the study could not eliminate the possibility that 
students who enrolled in I-BEST were more motivated or had other characteristics that may have 
distinguished them from students in other developmental courses . 10 For this reason, a 
randomized control trial of I-BEST is now in the planning stages by Abt Associates, with 
funding from the U.S. Department of Health and Human Services. 

Bringing Effective Program Strategies to Scale 

Though rigorous research on community college interventions is a relatively recent phenomenon, 
there are examples of institutions that are taking proven practices to scale. Kingsborough 
Community College offers a prime case: over several years, it has expanded its Learning 
Communities to serve two-thirds of entering freshmen and plans to grow the program further. 
Kingsborough’ s progress has been made possible by several factors, including strong 
commitment by its president; ongoing training and support for faculty who teach in learning 
communities; and a key funder, the Robin Hood Foundation, that has provided support over a 
sufficiently long period to help the college institutionalize the program using regular college 
revenues. Kingsborough recently won a grant from the U.S. Department of Education’s Fund for 
Improvement of Postsecondary Education to help other community colleges adapt its approach. 

The performance-based scholarship program in Louisiana did not continue beyond the 
demonstration, in part because the aftermath of Hurricane Katrina shifted the state’s attention to 
more pressing needs. Nonetheless, the positive findings prompted other states and institutions to 
develop versions of performance-based scholarships — some of which, as noted above, are 
undergoing further rigorous evaluation. If the findings are positive, the prospects for expansion 
are good. A considerable amount of public and private money already goes to financial aid 
programs, and some of these dollars may be designated for performance-based scholarships. 
Government or private sources may also create an incentive for institutions to implement the 
approach by offering start-up grants and matching funds. Finally, “how-to” guides and training 
for scholarship providers and financial aid administrators may encourage further take-up. 

The examples above underscore how credible evidence on programs can focus attention on an 
idea and create the will to bring effective programs to scale. At the same time, they show that 
running and testing a pilot project is only the beginning. States and colleges require strong 
leaders who can articulate a vision, set goals, and mobilize resources to continue building on the 
reforms. States and colleges must also pay attention to the faculty and staff responsible for policy 
and program implementation. Training and professional development are key; ultimately, many 
individuals — and not just a few champions — need to “own” the ideas and apply the research 
lessons to their own context. Finally, dependable funding is essential. 

States and colleges may benefit from two types of funding from government and private sources. 
One is for program innovation and testing, to continue the search for policies and programs that 
will lead to greater student success. A second is for adopting or expanding practices already 
proven to be effective. Of course, grant funding cannot be expected to last indefinitely and ought 
to include incentives or requirements for states and institutions to designate matching funds and 
develop long-term plans for sustainability. Evaluators can help by providing policymakers and 
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program administrators with information on costs and by analyzing the cost-effectiveness of 
strategies. For example, it may be possible to justify increased funding for interventions that 
accelerate time-to-degree or lead to other improved outcomes based on cost savings. 
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